Dynamic changes in task preparation in a multi-task environment: The task transformation paradigm.

A key element of human flexible behavior concerns the ability to continuously predict and prepare for sudden changes in tasks or actions. Here, we tested whether people can dynamically modulate task preparation processes and decision-making strategies when the identity of a to-be-performed task becomes uncertain. To this end, we developed a new paradigm where participants need to prepare for one of nine tasks on each trial. Crucially, in some blocks, the task being prepared could suddenly shift to a different task after a longer cue-target interval, by changing either the stimulus category or categorization rule that defined the initial task. We found that participants were able to dynamically modulate task preparation in the face of this task uncertainty. A second experiment shows that these changes in behavior were not simply a function of decreasing task expectancy, but rather of increasing switch expectancy. Finally, in the third and fourth experiment, we demonstrate that these dynamic modulations can be applied in a compositional manner, depending on whether either only the stimulus category or categorization rule would be expected to change.

Hierarchical control over foraging behavior by anterior cingulate cortex.

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

The successor representation subserves hierarchical abstraction for goal-directed behavior.

Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.

The controllosphere: The neural origin of cognitive effort.

Why do some mental activities feel harder than others? The answer to this question is surprisingly controversial. Current theories propose that cognitive effort affords a computational benefit, such as instigating a switch from an activity with low reward value to a different activity with higher reward value. By contrast, in this article, I relate cognitive effort to the fact that brain neuroanatomy and neurophysiology render some neural states more energy-efficient than others. I introduce the concept of the "controllosphere," an energy-inefficient region of neural state space associated with high control, which surrounds the better known "intrinsic manifold," an energy-efficient subspace associated with low control. Integration of control-theoretic principles with classic neurocomputational models of cognitive control suggests that dorsolateral prefrontal cortex (DLPFC) implements a controller that can drive the system state into the controllosphere, anterior cingulate cortex (ACC) implements an observer that monitors changes of state of the controlled system, and cognitive effort reflects a mismatch between DLPFC and ACC energies for control and observation. On this account, cognitive effort scales with the energetic demands of the DLPFC control signal, especially when the consequences of the control are unobservable by ACC. Further, I propose that neural transitions through the controllosphere lead to a buildup of neural waste. Cognitive effort therefore prevents against neural damage by discouraging extended periods of high control. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Alpha Event-Related Desynchronization During Reward Processing in Schizophrenia.

BACKGROUND: Alterations in the brain's reward system may underlie motivation and pleasure deficits in schizophrenia (SZ). Neuro-oscillatory desynchronization in the alpha band is thought to direct resource allocation away from the internal state, to prioritize processing salient environmental events, including reward feedback. We hypothesized reduced reward-related alpha event-related desynchronization (ERD) in SZ, consistent with less externally focused processing during reward feedback. METHODS: Electroencephalography was recorded while participants with SZ (n = 54) and healthy control participants (n = 54) played a simple slot machine task. Total alpha band power (8-14 Hz), a measure of neural oscillation magnitude, was extracted via principal component analysis and compared between groups and reward outcomes. The clinical relevance of hypothesized alpha power alterations was examined by testing associations with negative symptoms within the SZ group and with trait rumination, dimensionally, across groups. RESULTS: A group × reward outcome interaction (p = .018) was explained by healthy control participants showing significant posterior-occipital alpha power suppression to wins versus losses (p < .001), in contrast to participants with SZ (p > .1). Among participants with SZ, this alpha ERD was unrelated to negative symptoms (p > .1). Across all participants, less alpha ERD to reward outcomes covaried with greater trait rumination for both win (p = .005) and loss (p = .002) outcomes, with no group differences in slope. CONCLUSIONS: These findings demonstrate alpha ERD alterations in SZ during reward outcome processing. Additionally, higher trait rumination was associated with less alpha ERD during reward feedback, suggesting that individual differences in rumination covary with external attention to reward processing, regardless of reward outcome valence or group membership.

Electrophysiological measures of conflict and reward processing are associated with decisions to engage in physical effort.

Anterior cingulate cortex (ACC), a key brain region involved in cognitive control and decision making, is suggested to mediate effort- and value-based decision making, but the specific role of ACC in this process remains debated. Here we used frontal midline theta (FMT) and the reward positivity (RewP) to examine ACC function in a value-based decision making task requiring physical effort. We investigated whether (1) FMT power is sensitive to the difficulty of the decision or to selecting effortful actions, and (2) RewP is sensitive to the subjective value of reward outcomes as a function of effort investment. On each trial, participants chose to execute a low-effort or a high-effort behavior (that required squeezing a hand-dynamometer) to obtain smaller or larger rewards, respectively, while their brainwaves were recorded. We replicated prior findings that tonic FMT increased over the course of the hour-long task, which suggests increased application of control in the face of growing fatigue. RewP amplitude also increased following execution of high-effort compared to low-effort behavior, consistent with increased valuation of reward outcomes by ACC. Although neither phasic nor tonic FMT were associated with decision difficulty or effort selection per se, an exploratory analysis revealed that the interaction of phasic FMT and expected value of choice predicted effort choice. This interaction suggests that phasic FMT increases specifically under situations of decision difficulty when participants ultimately select a high-effort choice. These results point to a unique role for ACC in motivating and persisting at effortful behavior when decision conflict is high.

Task-level value affects trial-level reward processing.

Despite disagreement about how anterior cingulate cortex (ACC) supports decision making, a recent hypothesis suggests that activity in this region is best understood in the context of a task or series of tasks. One important task-level variable is average reward because it is both a known driver of effortful behaviour and an important determiner of the tasks in which we choose to engage. Here we asked how average task value affects reward-related ACC activity. To answer this question, we measured a reward-related signal said to be generated in ACC called the reward positivity (RewP) while participants gambled in three tasks of differing average value. The RewP was reduced in the high-value task, an effect that was not explainable by either reward magnitude or outcome expectancy. This result suggests that ACC does not evaluate outcomes and cues in isolation, but in the context of the value of the current task.

Interbrain synchrony: on wavy ground.

In recent years the study of dynamic, between-brain coupling mechanisms has taken social neuroscience by storm. In particular, interbrain synchrony (IBS) is a putative neural mechanism said to promote social interactions by enabling the functional integration of multiple brains. In this article, I argue that this research is beset with three pervasive and interrelated problems. First, the field lacks a widely accepted definition of IBS. Second, IBS wants for theories that can guide the design and interpretation of experiments. Third, a potpourri of tasks and empirical methods permits undue flexibility when testing the hypothesis. These factors synergistically undermine IBS as a theoretical construct. I finish by recommending measures that can address these issues.

Pain feedback interferes with reward positivity production.

The reinforcement learning (RL) theory of the reward positivity (RewP) proposes that RewP indexes a reward prediction error (RPE) signal processed in the anterior cingulate cortex (ACC). According to this theory, RewP is an event-related potential (ERP) that is more positive-going for feedback stimuli that predict better-than-expected outcomes (positive feedback) than for feedback stimuli that predict worse-than-expected outcomes (negative feedback). Despite strong evidence for this hypothesis, findings have been equivocal for tasks involving painful outcomes. We hypothesized that the RewP is modulated by high-level task goals such that outcomes that are congruent with the goals elicit positive RPEs even if their immediate consequences are negative. Accordingly, changes in high-level task goals should modulate RewP amplitude for tasks that involve seeking pain compared to tasks that involve avoiding pain. We recorded the electroencephalogram from participants who were instructed to navigate a virtual T-Maze to find reward-predictive feedback in a reward condition and pain-predictive feedback in a pain condition. We expected more positive-going ERPs to reward feedback in the reward condition and more positive-going ERPs to pain feedback in the pain condition. Despite behavioral results indicating that participants complied with task instructions, contrary to our predictions, we did not find a RewP to pain feedback. We suggest that pain feedback interfered with the effect of high-level task goals on RewP amplitude, which is indicative of conflict at different levels of task hierarchy.

Neural Representations of Task Context and Temporal Order During Action Sequence Execution.

Routine action sequences can share a great deal of similarity in terms of their stimulus response mappings. As a consequence, their correct execution relies crucially on the ability to preserve contextual and temporal information. However, there are few empirical studies on the neural mechanism and the brain areas maintaining such information. To address this gap in the literature, we recently recorded the blood-oxygen level dependent (BOLD) response in a newly developed coffee-tea making task. The task involves the execution of four action sequences that each comprise six consecutive decision states, which allows for examining the maintenance of contextual and temporal information. Here, we report a reanalysis of this dataset using a data-driven approach, namely multivariate pattern analysis, that examines context-dependent neural activity across several predefined regions of interest. Results highlight involvement of the inferior-temporal gyrus and lateral prefrontal cortex in maintaining temporal and contextual information for the execution of hierarchically organized action sequences. Furthermore, temporal information seems to be more strongly encoded in areas over the left hemisphere.

The Best Laid Plans: Computational Principles of Anterior Cingulate Cortex.

Despite continual debate for the past 30 years about the function of anterior cingulate cortex (ACC), its key contribution to neurocognition remains unknown. However, recent computational modeling work has provided insight into this question. Here we review computational models that illustrate three core principles of ACC function, related to hierarchy, world models, and cost. We also discuss four constraints on the neural implementation of these principles, related to modularity, binding, encoding, and learning and regulation. These observations suggest a role for ACC in hierarchical model-based hierarchical reinforcement learning (HMB-HRL), which instantiates a mechanism motivating the execution of high-level plans.

Hypnotic suggestions of safety reduce neuronal signals of delay discounting.

Waiting for delayed rewards is important to reach long-term goals, yet most people prefer immediate rewards. This tendency is called delay discounting. Evidence shows that people are more willing to wait for delayed rewards when they believe that the delayed reward is certain. We hypothesized that feeling safe makes delayed outcomes subjectively more certain, which should in turn reduce neuronal signals of delay discounting. We hypnotized 24 highly suggestible participants and gave them a suggestion to feel safe. We then used EEG to measure their brain responses to immediate and delayed rewards while they played a delayed gratification game. As compared to a control condition without hypnosis, participants that were suggested to feel safe under hypnosis reported feeling significantly safer. Further, their reward-related brain activity differentiated less between immediate and delayed rewards. We conclude that feeling safe makes delayed outcomes subjectively more certain and therefore reduces neuronal signals of delay discounting.

Beta oscillations following performance feedback predict subsequent recall of task-relevant information.

Reward delivery in reinforcement learning tasks elicits increased beta power in the human EEG over frontal areas of the scalp but it is unclear whether these 20-30 Hz oscillations directly facilitate reward learning. We previously proposed that frontal beta is not specific to reward processing but rather reflects the role of prefrontal cortex in maintaining and transferring task-related information to other brain areas. To test this proposal, we had subjects perform a reinforcement learning task followed by a memory recall task in which subjects were asked to recall stimuli associated either with reward feedback (Reward Recall condition) or error feedback (Error Recall condition). We trained a classifier on post-feedback beta power in the Reward Recall condition to discriminate trials associated with reward feedback from those associated with error feedback and then tested the classifier on post-feedback beta power in the Error Recall condition. Crucially, the model classified error-related beta in the Error Recall condition as reward-related. The model also predicted stimulus recall from post-feedback beta power irrespective of feedback valence and task condition. These results indicate that post-feedback beta power is not specific to reward processing but rather reflects a more general task-related process.

Reward processing electrophysiology in schizophrenia: Effects of age and illness phase.

BACKGROUND: Reward processing abnormalities may underlie characteristic pleasure and motivational impairments in schizophrenia. Some neural measures of reward processing show age-related modulation, highlighting the importance of considering age effects on reward sensitivity. We compared event-related potentials (ERPs) reflecting reward anticipation (stimulus-preceding negativity, SPN) and evaluation (reward positivity, RewP; late positive potential, LPP) across individuals with schizophrenia (SZ) and healthy controls (HC), with an emphasis on examining the effects of chronological age, brain age (i.e., predicted age based on neurobiological measures), and illness phase. METHODS: Subjects underwent EEG while completing a slot-machine task for which rewards were not dependent on performance accuracy, speed, or response preparation. Slot-machine task EEG responses were compared between 54 SZ and 54 HC individuals, ages 19 to 65. Reward-related ERPs were analyzed with respect to chronological age, categorically-defined illness phase (early; ESZ versus chronic schizophrenia; CSZ), and were used to model brain age relative to chronological age. RESULTS: Illness phase-focused analyses indicated there were no group differences in average SPN or RewP amplitudes. However, a group × reward outcome interaction revealed that ESZ differed from HC in later outcome processing, reflected by greater LPP responses following loss versus reward (a reversal of the HC pattern). While brain age estimates did not differ among groups, depressive symptoms in SZ were associated with older brain age estimates while controlling for negative symptoms. CONCLUSIONS: ESZ and CSZ did not differ from HC in reward anticipation or early outcome processing during a cognitively undemanding reward task, highlighting areas of preserved functioning. However, ESZ showed altered later reward outcome evaluation, pointing to selective reward deficits during the early illness phase of schizophrenia. Further, an association between ERP-derived brain age and depressive symptoms in SZ extends prior findings linking depression with reward-related ERP blunting. Taken together, both illness phase and age may impact reward processing among SZ, and brain aging may offer a promising, novel marker of reward dysfunction that warrants further study.

Smoking Decisions: Altered Reinforcement Learning Signals Induced by Nicotine State.

INTRODUCTION: Alterations in dopamine signaling play a key role in reinforcement learning and nicotine addiction, but the relationship between these two processes has not been well characterized. We investigated this relationship in young adult smokers using a combination of behavioral and computational measures of reinforcement learning. METHODS: We asked moderately dependent smokers to engage in a reinforcement learning task three times: smoking as usual, smoking abstinence, and cigarette consumption. Participants' trial-to-trial training choices were modeled using a reinforcement learning model that calculates separate learning rates associated with positive and negative prediction errors. RESULTS: We found that learning from positive prediction error signals is reduced during smoking abstinence and enhanced following cigarette consumption. By contrast, learning from negative prediction error signals was enhanced during smoking abstinence and reduced following cigarette consumption. Finally, when tested with novel pairs of stimuli, participants were relatively better at selecting the positive feedback predicting stimuli than avoiding the negative feedback predicting stimuli during the smoking as usual session, a pattern that reversed following cigarette consumption. CONCLUSIONS: These findings provide a specific computational account of altered reinforcement learning induced by smoking state (abstinence and consumption) and may represent a unique target for treatment of nicotine addiction. IMPLICATIONS: This study illustrates the potential of computational psychiatry for understanding reinforcement learning deficits associated with substance use disorders in general and nicotine addiction in particular. We found that learning from positive prediction error signals is reduced during smoking abstinence and enhanced following cigarette consumption. By contrast, learning from negative prediction error signals was enhanced during smoking abstinence and reduced following cigarette consumption. By highlighting important computational differences between three states of smoking, these findings hold out promise for integrating experimental, computational, and theoretical analyses of decision-making function together with research on addiction-related disorders.

Wearing a bike helmet leads to less cognitive control, revealed by lower frontal midline theta power and risk indifference.

A recent study claims that participants wearing a bike helmet behave riskier in a computer-based risk task compared to control participants without a bike helmet. We hypothesized that wearing a bike helmet reduces cognitive control over risky behavior. To test our hypothesis, we recorded participants' EEG brain responses while they played a risk game developed in our laboratory. Previously, we found that, in this risk game, anxious participants showed greater levels of cognitive control as revealed by greater frontal midline theta power, which was associated with less risky decisions. Here, we predicted that cognitive control would be reduced in the helmet group, indicated by reduced frontal midline theta power, and that this group would prefer riskier options in the risk game. In line with our hypothesis, we found that participants in the helmet group showed significantly lower frontal midline theta power than participants in the control group, indicating less cognitive control. We did not replicate the finding of generally riskier behavior in the helmet group. Instead, we found that participants chose the riskier option in about half of trials, no matter how risky the other option was. Our results suggest that wearing a bike helmet reduces cognitive control, as revealed by reduced frontal midline theta power, leading to risk indifference when evaluating potential behaviors.

Neural mechanisms of affective instability and cognitive control in substance use.

OBJECTIVE: We explored the impact of affect on cognitive control as this relates to individual differences in affective instability and substance use. Toward this end, we examined how different dimensions of affective instability interact to predict substance misuse and the effect of this on two event-related potential components, the reward positivity and the late positive potential, which are said to reflect the neural mechanisms of reward and emotion processing, respectively. METHODS: We recorded the ongoing electroencephalogram from undergraduate students as they navigated two T-maze tasks in search of rewards. One of the tasks included neutral, pleasant, and unpleasant pictures from the International Affective Picture System. Participants also completed several questionnaires pertaining to substance use and personality. RESULTS: A principal components analysis revealed a factor related to affective instability, which we named reactivity. This factor significantly predicted increased substance use. Individuals reporting higher levels of affective reactivity also displayed a larger reward positivity following stimuli with emotional content. CONCLUSION: The current study uncovered a group of high-risk substance users who were characterized by greater levels of affective reactivity and context-specific increased sensitivity to rewards. SIGNIFICANCE: These results help to elucidate the complex factors underlying substance use and may facilitate the creation of individually-tailored treatment programs for those struggling with substance use disorders.

Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex.

A longstanding view of the organization of human and animal behavior holds that behavior is hierarchically organized-in other words, directed toward achieving superordinate goals through the achievement of subordinate goals or subgoals. However, most research in neuroscience has focused on tasks without hierarchical structure. In past work, we have shown that negative reward prediction error (RPE) signals in medial prefrontal cortex (mPFC) can be linked not only to superordinate goals but also to subgoals. This suggests that mPFC tracks impediments in the progression toward subgoals. Using fMRI of human participants engaged in a hierarchical navigation task, here we found that mPFC also processes positive prediction errors at the level of subgoals, indicating that this brain region is sensitive to advances in subgoal completion. However, when subgoal RPEs were elicited alongside with goal-related RPEs, mPFC responses reflected only the goal-related RPEs. These findings suggest that information from different levels of hierarchy is processed selectively, depending on the task context.

What you give is what you get: Payment of one randomly selected trial induces risk-aversion and decreases brain responses to monetary feedback.

In economic studies, it is standard practice to pay out the reward of only one randomly selected trial (pay-one) instead of the total reward accumulated across trials (pay-all), assuming that both methods are equivalent. We tested this assumption by recording electrophysiological activity to reward feedback from participants engaged in a decision-making task under both a pay-one and a pay-all condition. We show that participants are approximately 12% more risk averse in the pay-one condition than in the pay-all condition. Furthermore, we observed that the electrophysiological response to monetary rewards, the reward positivity, is significantly reduced in the pay-one condition relative to the pay-all condition. The difference of brain responses is associated with the difference in risky behavior across conditions. We concluded that the two payment methods lead to significantly different results and are therefore not equivalent.

Brain mechanisms underlying apathy.

The past few decades have seen growing interest in the neuropsychiatric syndrome of apathy, conceptualised as a loss of motivation manifesting as a reduction of goal-directed behaviour. Apathy occurs frequently, and with substantial impact on quality of life, in a broad range of neurological and psychiatric conditions. Apathy is also consistently associated with neuroimaging changes in specific medial frontal cortex and subcortical structures, suggesting that disruption of a common systems-level mechanism may underlie its development, irrespective of the condition that causes it. In parallel with this growing recognition of the clinical importance of apathy, significant advances have been made in understanding normal motivated behaviour in humans and animals. These developments have occurred at several different conceptual levels, from work linking neural structures and neuromodulatory systems to specific aspects of motivated behaviour, to higher order computational models that aim to unite these findings within frameworks for normal goal-directed behaviour. In this review we develop a conceptual framework for understanding pathological apathy based on this current understanding of normal motivated behaviour. We first introduce prominent theories of motivated behaviour-which often involves sequences of actions towards a goal that needs to be maintained across time. Next, we outline the behavioural effects of disrupting these processes in animal models, highlighting the specific effects of these manipulations on different components of motivated behaviour. Finally, we relate these findings to clinical apathy, demonstrating the homologies between this basic neuroscience work and emerging behavioural and physiological evidence from patient studies of this syndrome.